System for introducing physical experiences into virtual reality (VR) worlds

ABSTRACT

A system for providing a user of a virtual reality (VR) system with physical interactions with an object in the real world or in the surrounding physical space while they are concurrently interacting in the virtual world with a corresponding virtual object. The real world object is dynamic with the system including a physical interaction system that includes a robot with a manipulator for moving, positioning, and/or orienting the real world object to move it into contact with the user. For example, the physical object is moved into contact with a tracked body part of the user at a time that is synchronized with a time of an interaction event occurring in the virtual world being created by the VR system. Further, a system is described for providing a dynamic physical interaction to a human participant, e.g., a fast and compelling handover in an augmented reality (AR) system.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part (CIP) of U.S. patentapplication Ser. No. 15/688,375, filed Aug. 28, 2017, which isincorporated herein by reference in its entirety.

BACKGROUND 1. Field of the Description

The present description relates, in general, to virtual reality (VR)systems and their uses to generate unique user experiences, and, moreparticularly, to systems and methods for providing the user of a VRsystem with physical experiences or haptic interactions while they areexperiencing a VR world or environment provided by a VR system.

2. Relevant Background

There is a growing interest in virtual reality (VR) games andapplications in the entertainment and education industries. VR typicallyrefers to computer technologies that use virtual reality headsets (orhead mounted displays (HMDs)) to generate realistic images, sounds, andother sensations that replicate a real environment or create animaginary setting. VR also simulates a user's physical presence in thisenvironment. VR has been defined as a realistic and immersive simulationof a three-dimensional 360-degree environment, created using interactivesoftware and hardware, and experienced or controlled by movement of thebody or as an immersive, interactive experience generated by a computer.

A person using virtual reality equipment is able to “look around” theartificial world, and, with higher quality VR systems, move about in itand virtually interact with features or items depicted in the headset. Avirtual world or environment (or its associated imagery) is displayedwith a virtual reality headset. VR headsets may include head-mountedgoggles with a screen in front of the eyes. Programs may include audioand sounds through speakers or headphones so that the user can hear thevirtual world, too.

Generally, though, the user can only “virtually” interact with objectsthey can see within the displayed virtual world of the headset andcannot touch or feel the virtual objects because the virtual objects donot exist in the real world. In some cases, advanced haptic systems, inwhich the VR user wears gloves, holds a game or other controller, and/orwears haptic clothing or a suit, may provide the user with some tactileinformation such as for use in medical, video gaming, and militarytraining applications. Similarly, some VR systems used in video gamescan transmit vibrations and other sensations to the user through thegame controller, but these feedback devices still do not allow a user tofeel and touch the virtual objects. Hence, the user experiences asensory inconsistency between what they can see and hear in the virtualworld and what they perceive with their sense of touch, and the VR usersare reminded or made aware of the artificial nature of any virtual worldprovided by existing VR systems.

SUMMARY

Briefly, a system is described herein that provides a user of a virtualreality (VR) system with physical interactions with an object in thereal world or in the surrounding physical space while they areconcurrently interacting in the virtual world with a correspondingvirtual object. The real world object is dynamic with the systemincluding a physical interaction system that includes a robot with amanipulator for moving, positioning, and/or orienting the real worldobject so as to move it into contact with the user (e.g., a tracked bodypart such as a hand, a tracked contact surface on the user's body, andso on) at a time that is accurately synchronized with a time of aninteraction event occurring in the virtual world being created by the VRsystem.

In creating the new system, the inventors recognized that physicalinteractions with dynamic objects in VR should be generated with precisecoordination and synchronization of the physical object and virtualobject counterpart in both location and timing. If either the positionor time of impact or physical interaction is not matched, the userexperiences discrepancies, which causes the illusion of the VR world tobe broken or degraded. To provide such coordination and synchronization,the system includes a robotic mechanism that is external to (and spacedapart from) the user in the space used for the VR experience, and therobotic mechanism is typically hidden from the user's view until theyplace the headset over their eyes.

The robotic mechanism (or physical interaction system) is controlled toselectively move the physical or real world object in the space. Thesystem includes devices for tracking one or more contact surfaces orbody parts (such as the hands) of the user during the VR experience. Acontrol program for the robotic mechanism can then use this trackinginformation to move the robotic mechanism's manipulator (e.g., a roboticarm with a gripper at its end) to keep the physical object out of theuser's reach (or not contacting the user's body at a tracked contactsurface or body part) until the very moment (defined by the VR systemfor an interaction event) when the object is precisely placed into theuser's hand (or other contacting surface on the user's body) with theappropriate orientation and velocity (speed and direction). While thevirtual object corresponding with the physical object may be flying,moving, or undertaking any arbitrary actions in the virtual world, it isonly the exact time of impact in the virtual world (time of aninteraction event) when the robot has to be controlled to place thephysical object at exactly the “same location” (e.g., in the user's handwhen a virtual object is placed in the user's virtual hand in thevirtual world by the VR system).

Using a robotic mechanism to control the physical object provides thesystem with the necessary level of precision. However, use of a roboticmechanism also enhances safety as the robotic mechanism can be operatedto actively remain a distance outside a predefined safety envelope aboutthe user's body until the appropriate time (move the manipulator arm toprovide physical interaction with the object at the time defined for theinteraction event). The robotic mechanism's controller may also monitorand control interaction forces with the VR system user to prevent unsafecollisions. The programmable nature of the system further allows otherimpulsive effects such as snatching the physical object from the user'shand at an appropriate time and location. In other cases, theinteraction event is an extended event such as a pushing/pulling on theuser (on or at the user's contact surface), emulating the breaking ofthe object, or any variable dynamic interaction that is synchronizedbetween the physical and virtual worlds.

More particularly, a system is provided that is adapted for providing adynamic physical interaction to a person during a virtual reality (VR)experience. The system includes a VR system including a headset with adisplay screen and a VR rendering module generating a video output. Thedisplay screen displays an image of a virtual world based on the videooutput. The system also includes a physical interaction systemcomprising a robotic mechanism with an object manipulator and a robotcontrol module generating commands for operating the robotic mechanismto selectively position a physical object within a space. Duringoperations, a wearer of the headset is positioned in the space, and theimage of the virtual world includes an image of a virtual objectcorresponding to the physical object. Further, the commands generated bythe robot control module cause the robotic mechanism to move thephysical object in the space based on a state of the virtual object inthe virtual world.

In some preferred embodiments, the commands generated by the robotcontrol module cause the object manipulator to move the physical objectinto contact with a surface on a body part of the wearer at a firsttime. The image of the virtual world includes an image of a virtual bodypart corresponding to the body part of the wearer. Further, the image ofthe virtual object is shown in the image of the virtual world to moveinto contact with the image of the virtual body part at a second timematching the first time.

In these embodiments, a tracking system can be included that generatestracking data for the body part of the wearer. Then, the image of thevirtual body part is provided at a location in the image of the virtualworld corresponding to a body part location identified by the trackingdata, and the object manipulator moves the physical object to a locationin which the physical object at least partially coincides with the bodypart location to cause the physical object to contact the body part ofthe wearer at the first time. Further, the tracking data includes anorientation of the body part of the wearer, and the commands cause theobject manipulator to adjust an orientation of the physical object basedon the orientation of the body part of the wearer prior to the firsttime.

In some cases, the state of the virtual object in the virtual worldincludes a velocity. Then, the commands can cause the object manipulatorto move through the space at a velocity matching the velocity of thevirtual object in the virtual world. In these or other cases, thecommands generated by the robot control module cause the objectmanipulator to release the physical object at a third time after thefirst time. In some embodiments, the body part includes a hand of thewearer of the headset, the surface is the palm of the hand, and thecommands cause the object manipulator to halt travel at the location orat a preset distance past the location at about the first time and/orafter a predefined contact force is imparted or detected by the controlmodule of the robotic mechanism.

In another useful implementation, a system is described for providing adynamic physical interaction to a human participant (e.g., a fast andcompelling handover in an AR system). The system includes an augmentedreality (AR) system displaying or projecting augmentation contentvisible by a participant in a space. The system further includes aphysical object positioned in the space, and a physical interactionsystem. In one embodiment, the physical interaction system includes arobot with an object manipulator and a controller generating controlsignals to operate the robot. The control signals cause the objectmanipulator to perform a handover of the physical object to or from theparticipant. The object manipulator moves toward or away from anexpected handover location in the space at a speed matching or less thana predefined handover speed for a human performing handovers.

In some of these systems, the controller initiates the handover apredefined delay period after sensing initiation of the handover by theparticipant. The predefined delay period may be set at sensorimotorreaction time for a human performing handovers. In these and otherembodiments, the control signals cause the object manipulator to travelover a predefined smooth trajectory toward the expected handoverlocation, and the controller modifies the travel of the objectmanipulator prior to the handover based on tracking data for thephysical object, whereby the object manipulator converges on an actuallocation of the physical object in the space.

In these and other implementations of the system, the handover is arobot-to-participant handover, and the controller generates the controlsignals to cause the object manipulator to release the physical objectonly after sensing of a force greater than a minimum force threshold isbeing applied upon the physical object. In some cases, the handover is aparticipant-to-robot handover, and the controller generates the controlsignals to cause the object manipulator to return to a rest pose onlyafter grasping the physical object and sensing the participant hasreleased the physical object. In some preferred embodiments, theaugmentation content is superimposed upon at least the objectmanipulator of the robot to modify appearance of the robot to theparticipant or to at least partially block or disguise viewing of therobot in the space by the participant. The system may also include atracking system tracking a location of the physical object in the space,and the augmentation content can be superimposed upon the physicalobject to visually augment the physical object as the physical object ismoved about the space based on the tracked location.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram of a system for providing a user ofa VR system with physical experiences that are synchronized with eventsor activities being generated in a VR world or environment by the VRsystem;

FIG. 2 illustrates a VR space in which a VR user is being prepared for aVR experience by a human VR system operator/guide and showing a physicalobject for interaction by the VR user during operations of a systemsimilar to that shown in FIG. 1;

FIG. 3 illustrates positioning of the physical object upon a physicalstand/support structure prior to the VR user entering the VR world;

FIG. 4 illustrates an initial operating step of a system as shown inFIGS. 1-3 in which a VR user interacts with a displayed VR world;

FIG. 5 illustrates a step that may be performed concurrently with thestep shown in FIG. 4 in which a visual shield is moved to expose, andallow full operations of, a robotic mechanism;

FIG. 6 illustrates a next step in the operation of the system shown inFIGS. 2-5 showing the robotic mechanism of the physical interactionsystem and an operator placing the physical object in the robot'sgripper (or attaching it to the robot's object manipulator);

FIG. 7 illustrates operation of the system of FIGS. 2-6 while the VRuser interacts with a VR world provided by the VR system and thephysical interaction system concurrently acts to retain the physicalobject properly oriented relative to the VR user's hand but at apredefined spacing;

FIG. 8 illustrates a display screen/monitor of the VR headset of thesystem during the operations or step shown in FIG. 7;

FIG. 9 illustrates a step or operations of the system when the physicalinteraction occurs, and the interaction event concurrently occurs in theVR world provided by the VR system;

FIG. 10 illustrates the display screen of the VR headset of the systemduring operations of the VR system to update views of the VR world aftera physical object has been placed into the VR user's hand;

FIG. 11 shows the VR space in a later step or operations of the systemto hide the presence of the physical interaction system and while the VRuser continues to manipulate the physical object to affect the VR worldexperience provided by the VR system;

FIG. 12 illustrates a functional block diagram of another implementationof a system of the present description useful for providing physicalinteractions to a user of VR system that are synchronized withinteraction events in the VR world provided by the VR system;

FIG. 13 illustrates a side view of a robot useful in a fast handoversystem of the present description with an object manipulator in the formof a robotic arm with a hand with operable fingers for grasping aphysical object;

FIGS. 14A-14F illustrate a handover sequence as carried out by ahandover system that includes the robot of FIG. 13;

FIG. 15 illustrates a physical object as used in testing the handoversystem of the present description in a support cradle prior to and afterhandover operations;

FIG. 16 is a front view of the robot of FIG. 13 showing more detail withthe robot in an at rest pose;

FIGS. 17A-17C show a process of calculating the online joint trajectoryfor a robot (or an object manipulator of a robot as shown in FIG. 1)with schematics and graphs;

FIG. 18 is a human participant-to-robot handover method of the presentdescription; and

FIG. 19 is a robot-to-human participant handover method of the presentdescription.

DETAILED DESCRIPTION

Prior to turning to specific examples and/or implementations, it may beuseful to discuss more generally the components and processes that maybe used to provide a user of a VR system with a haptic experience toincrease the quality of the VR world or experience (or to make the VRexperience “more real”). FIG. 1 illustrates a functional block diagramof a system 100 for inserting haptic experiences into a VR world for auser 104 of a VR system 110. Particularly, during operations of thesystem 100, a user 104 of a VR system 110 is able to have physicalinteractions 166 with a physical or real world object 160. The object160 has one or more contact surfaces 162 that are oriented so as to matewith a tracked body part and/or contact surfaces on the body of the user104 during or just prior to the physical interaction 166 as shown witharrows 164 by the physical interaction system 170 (e.g., by a robot 178and its object manipulator 179 in the system 170). The direction oftravel toward and away from the user's body part/contact surfaces aswell as the velocity of the object 160 is also controlled and/orprovided by operations of the physical interaction system 170 as shownwith dashed line 167 and also with arrows 164 (e.g., the object 160 maybe moved in any direction and at nearly any desired speed to provide adesired physical interaction 166 with the body part/contact surfaces ofthe user 104).

The VR system 110 includes a processor 112 that executes code orprogramming to provide the functionality of a VR suite or application114. The VR system 110 includes a headset (or headgear/eyewear) 120 thatis worn by the user 104 during operations of the system 100. The headset120 includes a display or monitor 122 that is operated by the processor112 and VR suite/application 114 to display imagery or images generatedby a virtual world generator/engine of the VR suite/application 114.While such images are displayed, the user 104 typically has their visionof the surrounding physical environment or space blocked and can onlysee the VR images or VR world/environment provided by the VRsuite/application 114. The VR system 110 further includes input/output(I/O) devices 124 for allowing a user 104 to initiate and end operationsof the VR system 110 (e.g., to start or stop the VR suite/applicationand its creation of a VR experience). The I/O devices 124 may includewired or wireless communication devices for transmitting messages 125 toI/O devices 174 of the physical interaction system 174, and suchcommunications 125 are typically used to share tracking data for user104 (see data 194 in memory 180) and also to provide synchronization (intime and location) of a VR event/activity with a real world movement bythe robot 178 of an object 160 to provide a desired physical interaction166 with the user 104.

The VR system 110 includes or has access to memory 130 (with accessmanaged by processor 112), and the memory 130 stores an interactionevent definition 132. For example, it may be desirable for a user thatis experiencing a virtual world created by the generator/engine 116 toalso have a physical interaction 166 to make the virtual world morelife-like. To this end, the event definition 132 may define particularactions by the user 104 (e.g., particular movements of one or moretracked body parts of the user 104, particular movement of the user 104within the virtual world, and the like) that will trigger the physicalinteraction 166, and the event definition 132 may also define which oftwo or more physical objects 160 are to be positioned relative to theuser 104 for the physical interaction 166 as well as the orientation andvelocity 164 of the object 160 at the time of the physical interaction166 (at the time set by the interaction event definition 132 and/or ascalculated by the control program 176 as shown at 182).

For example, the interaction event definition 132 may call for the user104 to catch or kick a ball when they place their hand or foot in aparticular location in a virtual world or space with a particularorientation of contact surfaces, and the physical interaction system 170operates to place the object 160 with its contact surfaces 162 in theproper real world space about the user 104 at the time the event 132 inthe virtual world takes place. In another example, a hilt or handle of atool or weapon that is visually presented by the generator/engine 116 inthe virtual world may be provided as a physical object 160 that isplaced in the hand of the user 104 with a particular orientation and ata desired speed as shown at 164 to provide the physical interaction 166at a time that is synchronized with an occurrence of the interactionevent 132 in the virtual world generated by engine 116. In both theseexamples, the user 104 is observing images of virtual objects on thedisplay/monitor 122 of the headset 120 when the interaction event 132and physical interaction 166 concurrently occur in time so that a hapticfeedback or experience is inserted into the VR experience/environmentprovided by the VR system 110.

To provide such functionality, the VR system 110 includes a trackingsystem 150 combined with tracking elements (such as a tracking glove)152 that are worn by the user 104. Particularly, the user 104 wears theelements 152 on or near contact surface or on a body part that is thetarget for the physical interaction 166 (such as the palm of one or bothof the user's hands, the user's head, the user's foot, the user'schest/torso, and so on) with the physical object 160. The tracking data153 is collected and processed by the tracking system 150 to determineor generate a set of tracked data 140 that is stored in memory 130, andthis may include a location and, in some cases, orientation of a bodypart (e.g., a hand) of the user 104 (or of the contact surfaces for thatbody part). The virtual world generator/engine 116 may be configured togenerate an image of the tracked body part for the virtual world asshown at 142 in memory 140, and this image 142 would be presented to theuser 104 via imagery displayed on the display/monitor 122 of theheadset. For example, the user 104 may be able to see an image of theirhand or other monitored body part in the virtual world, and, due to thetracking by system 150, the virtual hand/body part's image 142 moves inthe virtual world with orientation and speed that matches that of theirphysical hand/body part. The interaction event definition 132 can beadapted to trigger the physical interaction 166 when the user's trackedbody part/contact surface is properly positioned at a particular pointalong a story timeline for the virtual world.

A definition of a virtual object 134 is stored in memory 130, and thegenerator/engine 116 uses this definition to insert an image of thevirtual object 134 that corresponds (exactly or loosely) with thephysical or real world object 160 into the virtual world/environmentbeing provided to the user 104 on the headset display/monitor 122. Thevirtual object 134 may have a state or status 136 defined for it at eachpoint in time in the virtual world/reality, and the state 136 mayinclude the location 137 of the virtual object 136 in the virtual worldor space (which allows calculation of a distance between the virtualrepresentation of the user's body part/contact surface and the virtualobject in the virtual world), the orientation 138 of the virtual object136, and the present velocity 139 of the virtual object 136 in thevirtual world (which allows determination of when the physicalinteraction 166 should occur which typically coincides with a definitionof an interaction event 132).

In some embodiments, the physical object 160 may be visible to the user104 prior to their placing the headset 120 over their eyes. For example,the system 100 may be operated to establish a virtual world storylinethat involves the physical interaction 166 between the user 104 and thephysical object 160, e.g., the user 104 is shown a ball 160 and toldthey will be catching, throwing, grabbing, and/or kicking the ball 160in the virtual world, the user 104 is shown a tool or weapon (e.g., amedieval or futuristic sword) with a handle or hilt that they are toldthey will grab or receive in their hands for use in the virtual world,and so on. At this point, all or portions of the physical interactionsystem 170 may be hidden from view of the user 104 via a visualshield/barrier 168, and, after the user 104 places the headset 120 on,the shield 168 may be repositioned to allow the system 170 (e.g., therobot 178) to access the object 160. This may be achieved automaticallywith a movement assembly (not shown) or manually by an operator 108. Theoperator 108 may also reposition the object 160 from this initialstoryline position into a predefined position for manipulation by therobot 178 such as in the gripper of the object manipulator 179 of therobot 178, on a stand for retrieval by the robot 178, and the like.

The physical interaction system 170 is included in the system 100 toselectively move and position 164 the physical object 160 to provide thephysical interaction 166 with the user 104 at a time and manner(orientation and speed and direction of movement) that is synchronizedor matched with an interaction event 132 in the virtual world generatedby the VR system 110. To this end, the physical interaction system 170includes a processor 172 that manages I/O devices 174 to receivecommunications 125 from the VR system 110, and these communications 125may include an interaction event definition 181 that the processor 172stores in memory 180. This definition 181 may including definitions ofmovements for a robot 178 to properly position/move 164 the physicalobject 160 to synchronize the physical interaction 166 with the event132 in the virtual world provided to the user 104 by VR system 110. Acontrol program 176 is run/executed by the processor 172 to process theinteraction event definition 181 and to generate control signals for therobot 178 to achieve the desired movements/positioning 164 of the object160 with synchronization with the VR system 110.

The physical interaction system 170 includes a robot 178 to selectivelymove and position/orient 164 the object 160 to provide its contactsurfaces 162 in a desired location relative to the user's tracked bodyparts/contact surfaces. The term “robot” is used in its broad sense toinclude nearly any electro-mechanical machine that can be operated toperform the functions described herein including supporting, gripping,and/or holding an object 160 and moving it to a desired location with aparticular velocity and in a desired direction. The robot 178 typicallywill include an object manipulator 179 to perform these functions. Theobject manipulator 179 may take the form of a robotic arm with agripper/hand at one end that can grab/receive the object 160, and theobject manipulator 179 may include motors and/or servos to move the arm(and its joints) to selectively move the hand/gripper with the object160 to provide the movement/positioning/orienting 164 of the contactsurfaces 162 of the object to provide the desired physical interaction166 (e.g., place the object 160 against or in the tracked bodypart/contact surfaces of the user 104 at a particular time that matchesoccurrence of an interaction event 132 in the virtual reality providedto the user 104 by the VR system 110).

The control program 176 is configured to compute or calculate a time 182for an upcoming interaction event 132 in the virtual world presentlygenerated by the VR system 110. Then, in response, the control program176 generates control signals to the robot 178 to operate the objectmanipulator 179 to place/move 164 the object 176 to time the physicalinteraction 166 to occur at this time 182. To allow this calculation,the system 170 and/or control program 176 is configured to gather andstore the following data in memory 180: (a) an object state 184 thatdefines a location 185 of the object 160 in a space in which the user104 is located that may be determined by knowledge of the known locationof the object manipulator and/or via a second tracking system (such assystem 150) along with tracking elements provided upon the object 160(or its contact surfaces 162) (from which a relative location of theobject 160 to the tracked body part/contact surface of the user 104 canbe determined), an orientation 186 of the object 160, and a currentvelocity 187 (direction and speed) of the object 160; (b) a virtualobject state 190 that defines (from communications 125 with the VRsystem 110) a velocity 191 of the virtual object 136 that correspondswith the physical object 160 in VR world/space, a location 192 of thevirtual object 136 in this VR world/space (e.g., relative to a virtualrepresentation of the user 104 in the VR world/space (or their trackedcontact surfaces with elements 152), which may be used to determine thetime 182 for the occurrence of the interaction event 132 and velocityand movement 164 that should be imparted to the object 160 by the robot178), and an orientation 193 of the virtual object 136 (which can beused to determine if the robot 178 should be controlled to move/reorient164 the object 160); and (c) tracked user data 194 from the trackingsystem 150 via communications 125 that may include a location 195 of auser's body part or intended contact surfaces on the body of the user124 for the physical interaction 166 and also include an orientation 196of the user's body part/contact surfaces in the physical space thatincludes the system 170 (and, particularly, the user 104, the object160, and the robot 178). When the interaction 166 is completed, thevisual shield 168 may be returned to its original position to block theuser 104 from viewing the physical interaction system 170 (or at leastthe robot 178) when they remove the headset 120, thereby furthering adesired illusion if useful in some implementations of system 100.

With system 100 understood, it may now be useful to describe withreference to figures one exemplary implementation of the system 100during its use to provide a physical interaction to a user of a VRsystem. In this exemplary implementation, a VR user is instructed thatin the VR world they are participating in or in which they will beimmersed that a tool or weapon (such as an energy sword or saber that isthe physical object in this example) will be made available for theiruse. In the VR world, they will use their superpower of telekinesis toconcentrate on moving the tool/weapon, and, when they concentrate withenough energy, the tool/weapon will fly through the air and land (withappropriate impact forces associated with the size/weight and velocityof the flying object) in their hand. In practice, the VR world iscontrolled to generate a VR representation of the user's hand(s) as theyreach out to an image of the object in the VR world (e.g., the virtualobject), and, via communications with the controller of the roboticmechanism, the physical interaction system operates to follow the user'shand position, to orient and position the tool/weapon (hilt ofsword/saber or handle of a tool/weapon for use in the VR world) relativeto the user's hand (tracked by the VR system), and to move thetool/weapon (hilt, handle, or the like) into the user's outstretchedhand at the moment when the interaction event occurs in the VR world.

With this VR world/physical interaction storyline in mind, FIG. 2illustrates an initial operational step for the system (e.g., for system100 of FIG. 1) that sets up the story and expectations of the VR user.As shown, a VR user 204 is in a VR/interaction space 202 in which thesystem with its VR system and physical interaction system are positioned(such as systems 110 and 170 of FIG. 1 and not visible in FIG. 2). Ahuman operator/guide 206 is also present in the space 202, and theoperator/guide 206 explains the VR/interaction storyline to the VR user204 (e.g., that they need to use their special powers to cause aweapon/tool to fly through the air to their hand for use in the VRworld). The operator/guide 206 also shows them the physical object 260that they will be interacting with or manipulating while in the VRworld. In FIG. 3, a next process (or system operation or pre-operation)step is shown in which the human operator/guide 260 is acting to placethe tool/weapon (physical object) 260 within a case/display 305 upon astand 306 (both of which are physical objects in the VR space 202).

In FIG. 4, the system has begun initial operations with the VR user 204still located in the VR space 202 and with the user 204 shown to bewearing a VR headset 420 on their head 207 and a tracking glove 450 ontheir hand 205. The system (e.g., the VR system) includes a trackingsystem (such as system 150 in FIG. 1) that operates to track thelocation of the user's hand 450 by monitoring movement/location oftracking elements/buttons on the glove 450. Further, the tracking systemmay operate to track the position/orientation of the user's head 207 bytracking location/movement of tracking elements/buttons 452 on the VRheadset 420, and the head position/orientation may be used by the VRsystem to determine the VR user's line of sight in the VR world.

The headset 420 includes a display screen/monitor, which is shown at 422for ease of explanation, and the screen 422 is operated by the VR systemto display a view/image 423 of the virtual world/environment in whichthe VR user 204 is being immersed during system operations. The VRsystem operates to display an image 434 of the virtual object (e.g., arepresentation of the physical object 260 in the form of a weapon/toolin the virtual world 423), and the virtual object image 434 may bepositioned within the virtual world 423 at a location that correspondswith the base/stand 306 in the VR space 202 relative to the location ofthe VR user 204. The VR user 204 now moves their hand 205 about in theVR space 202, and this location (and hand orientation) is tracked by thetracking system, and such tracking information is used by the VR systemto generate and locate an image 442 of the user's hand in the VR world423 (e.g., to display an image of the tracked body part or interactionsurface on the VR user's body). In this storyline, the user 204 isreaching out their hand 205 in the VR space 202 which results in theimage 442 of their virtual hand (or tracked body part) also moving andextending outward toward the image 434 of the virtual tool/weapon in thedisplayed VR world 432 on the headset screen 422.

Concurrently with the step shown in FIG. 4, as shown in FIG. 5, theoperator/guide 206 may move in the VR space to the physical interactionsystem to set it up for next operations of the system. Particularly, theoperator/guide 206 is shown to be positioned next to a stand/base 567for a robotic mechanism, which was hidden from view by the VR user 204by the cover or visual shield 568. The operator/guide 206 in FIG. 5 isshown to be removing or lifting 569 the cover/shield 568 to expose therobotic mechanism and allow it to generate the physical interaction withthe physical object (weapon/tool 260).

FIG. 6 illustrates the VR space 202 with the cover/shield 568 fullyremoved from the base/stand 567. As shown, the system includes a roboticmechanism 678 with an object manipulator 679 in the form of a movablearm with a gripper at its end. In the operating step shown in FIG. 6,the VR user 204 is still wearing the VR headset 420 and is interactingwith the VR world displayed on the screen of the headset 420 (and cannotsee out into the VR space 202 such that the user 204 cannot see therobotic mechanism 678 or actions by the operator 206 in the step shownin FIG. 6). In this step, the operator/guide 206 is acting to move thephysical object (weapon or tool in this example) 260 from the stand 306to the robotic mechanism 678. Particularly, the operator 206 is placingthe object 260 in the object manipulator 679 (e.g., in the gripper ofthe robotic arm). In other embodiments, the uncovered robot may act toretrieve the object itself as an initial step or it may already beholding an object matching or similar to the object 260 (and the removalof the object 260 from the stand 306 may be manual or automated in suchimplementations of the system).

In FIG. 7, the VR space 202 and system is shown in a later operationalstep. Particularly, the VR user 204 is interacting with a VR world beinggenerated by the VR system including operation of headset 420 to displayimages of the VR world including an image of the VR object correspondingwith the physical object 260. The VR user 204 is shown to be movingtheir hand 205, which has its location and orientation tracked viatracking element glove 450 (and tracking system of the VR system), andthis results in the user's hand 205 being at one or more locations inthe space 202. The physical interaction system communicates with the VRsystem to obtain the hand location/orientation and also to obtain statusof the VR world storyline (e.g., location and orientation and velocityof the virtual object relative to the user's hand in the VR world).

In response, the robotic mechanism 678 is operated so as to move theobject manipulator 679 (e.g., arm with gripper) so as to move, position,and orient the physical object 260 as shown with arrows 779 in 3D spacerelative to the user's hand 205. The robotic mechanism 678 is operatedso as to maintain the object 260 (e.g., a weapon/tool or just itshandle/hilt) a predefined distance, d, apart from the user's hand 205(or a contact surface of this tracked body part such as the palm of thehand 205). In some cases, this distance is 6 to 24 inches to allowprompt delivery of the object 260 but also minimize risk of inadvertentcontact with the user 204 prior to an interaction event in VR world withthe corresponding virtual object. The object 260 typically has itsorientation changed over time by the manipulator's motion 779 so as tomirror either the orientation of the corresponding virtual object or toprepare it for proper mating with the contact surface of the trackedbody part (e.g., to remain ready for movement of a particular contactsurface/area of the object 260 into the palm of the user's hand 205) toobtain a physical interaction that matches that planned for the VRworld.

The process step/operation of the system shown in FIG. 7 continues untilthe physical interaction event is triggered in the VR system (or apredefined time before the planned event to allow the robotic mechanism678 to have adequate time to move the object 260 across the separationdistance, d, at a desired velocity to provide the physical interactionand/or contact of the object 260 with the hand 205). FIG. 8 illustratesthe screen/monitor 422 in the headset 420 during the step or operationsof FIG. 7 of the system. As shown, the VR system is operating to presentan image 423 of a VR world in which the user's hand 442 is displayed andis oriented and located similar to the user's hand 205 in the space 202.The VR world 423 includes an image of the virtual object 434corresponding with the physical object 260, and the user's virtual hand442 is reaching outward toward the object image 434 with their palmexposed so as to be ready to receive the object 434. Such a predefinedpositioning combined with proper orienting of the hand 205 (and, hence,virtual image 442) combined with timing and other parameters/actions inthe VR world provided by the VR system may act as the “trigger” for aninteraction event.

Once an event is triggered, the controller of the robotic mechanism 678acts to determine a time until occurrence of the event in the VR world.The controller then acts to move the manipulator 679 at a time beforethe event occurrence time that is chosen for a predefined deliveryvelocity to cause the object 260 to be delivered into the user's hand205 in a time synchronized manner at the delivery velocity and with adesired orientation (e.g., one matching that of the user's hand and/orof the corresponding virtual object in the VR world).

FIG. 9 illustrates operation of the system during the physicalinteraction.

Particularly, the robotic mechanism 678 is operating via movement of itsmanipulator (arm and gripper in this case) 679 to position the physicalobject 260 into the VR user's hand 205 (which is covered by trackingelement glove 450). In the VR world provided by the VR system, the VRworld is configured to include a defined interaction event that is timedto occur at the time when the object 260 contacts the hand 205 (i.e.,the tracked body part or contact surface on the user's body). In otherwords, the VR world displayed to the user on the display screen nowshows the virtual object in the displayed image of the user's hand(e.g., after being shown flying or otherwise traveling from a firstlocation in the VR world to a second location in the VR worldcorresponding with the user's hand location). In this way, the timing ofthe physical interaction and the interaction event in VR world are timesynchronized.

In this example, the physical interaction is defined as the robot 678placing the object 260 in the user's hand 205 (or against the palm) at aparticular velocity (and/or with a particular contact force) and thenfor the manipulator to release the object 260 and move away from theuser 204. At this point, the user 204 is solely holding and supportingthe physical object 260. In other examples, though, the roboticmechanism 678 may continue to hold/support the object such as to providethe feel of a struggle for the object 260, to provide resistance torelease the object (e.g., pull an object away from a VR world surface orcharacter), or to even pull the object 260 back away from the user'shand 205 (or other contact surface on the user 204).

FIG. 10 illustrates the display screen/monitor 422 of the user's headset420 as it is being operated by the VR system after the physicalinteraction and interaction event shown in FIG. 9. As shown, the VRsystem acts to now show that the user's hand is holding the physicalobject with an image 442 in the virtual world 423 of the users' hand (ortracked body part) generated to show it grasping or holding the weaponwith an image 434 of the virtual object corresponding to the physicalobject 260 shown in the virtual hand/image 442. The virtual version 434of the physical object 260 can now be used/activated in the VR world 423such as by further tracking of the user's hand via tracking elementglove 450 and tracking components of the VR system.

In some cases, tracking elements/buttons may also be provided on thephysical object 260 to allow the VR system to track itslocation/orientation or to determine if/when the user 204 stops holdingit in their hand 205 (or if they unsuccessfully catch or grasp theobject 260 in the interaction step shown in FIG. 9). The trackinginformation may also or instead be used by the physical interactionsystem to control movements of the manipulator 679 to facilitate properorienting of the object 260 prior to the handoff/interaction provided inFIG. 9.

FIG. 11 illustrates the VR space 202 after the step/operation of thesystem shown in FIGS. 9 and 10. Particularly, the VR user 204 is shownto be still grasping the physical object 260 in their hand 205 coveredby tracking element glove 450. The VR system operates to modify the VRworld it creates and displays to the user in the screen of the headset420 to reflect the presence of the object 260 (or of a virtual object inthe VR world that corresponds to the physical object 260). To furtherthe VR experience and physical world interaction for the user 204, theoperator/guide 206 acts, while the user 204 is still wearing the headset420 such that their ability to view the space 202 is prevented, toreturn the cover/shield 568 to its original location to hide thepresence of the robotic mechanism 568 from the user 204. After the VRexperience is completed, the VR user 204 can remove the headset 420, andthe VR space 202 will appear consistent with their expectations with thephysical object 260 in their hand and the visual shield/cover 568 backin place.

FIG. 12 illustrates a functional block diagram of another implementationof a system 1200 of the present description useful for providingphysical interactions to a user of a VR system that are synchronizedwith interaction events in the VR world provided by the VR system. Thesystem 1200 may include components similar to or different from those ofthe system 100 and may be operated to provide the VR experience withphysical interactions as shown in FIGS. 2-11.

As shown, the system 1200 is operated to provide a unique VR experiencewith physical interactions for a VR user 1204. The user 1204 has thelocation, movement, and/or orientation of their hand(s) 1205, otherlimbs/body parts/contact surfaces 1206, and/or head 1207 tracked bycollection of tracking data 1262 such as including information 1264 fromtracking elements on the hand 1205, limbs 1206, and head 1207 by atracking system 1260. The tracking system 1260 may be configured forexternal sensing such as with motion capture, with an IMU (inertialmeasurement unit), or the like. This tracked information is processedand provided to the VR rendering module 1210 and robot control module1230 as shown at 1268 for their use, respectively, in generating a VRworld and in controlling a robot 1240 to provide desired physicalinteractions with the user 1204.

The VR system portion of the system 1200 includes a VR rendering module1210 (such as a VR application available from Unity or the like), whichgenerates a video output 1215 (images of a VR world) in part based ontracking data 1268 of the user 1204 as well as of the physical object1250. The video output 1215 is provided to a head mounted display (HMD)1220 that is worn by the user 1204 on their head 1207, and the HMD 1220includes a screen for providing a display 1225 generated from the videooutput 1215 from the VR rendering module 1210. As discussed above, theVR rendering module 1210 is configured to display virtual images of theuser 1204 such as of their hand 1205 or other limbs 1206 that are to becontact surfaces for a physical interaction with a physical object 1250.The module 1210 also displays an image of a virtual object in the videooutput 1215 that corresponds to the physical object 1250 and that iscaused to be located, oriented, and moved in the VR world provided inthe video output to suit a particular VR world storyline and to supportone or more interaction events.

The physical interaction system components of the system 1200 include arobot control module 1230, which may include software for a robotinterface. The module 1230 is configured to receive as input VR stateinformation 1218 from the VR rendering module 1210 that may define alocation of a virtual object (corresponding to the physical object 1250)in the VR world being provided in display 1225 to the user 1204. The VRstate information 1218 may also include the present velocity of thevirtual object (again, corresponding to the physical object 1250) in theVR world associated with the video output 1215 and provided in display1225 to the user 1204. Additional information in the VR stateinformation may include the orientation of the VR object and,significantly, a definition of an upcoming interaction event in the VRworld (e.g., when it may be triggered and the like) for use insynchronization of movement of the object 1250 to a position relative tothe user 1204 to provide a haptic 1255 or physical interaction with theobject 1250.

The robot control module 1230 further receives as input tracking data1268 from the external sensing-based tracking system 1260. The trackingdata 1268 may include the present location, velocity, and/or orientationof the object 1250 (and/or contact surfaces on the object 1250) and mayalso include the tracking data from the user 1204 (such as a location ofone or more contact surfaces on the hand 1205, the limbs 1206, and thehead 1207 or the like). The robot control module 1230 may facilitatesynchronization of the VR world (or video output 1215) with operation ofa robot 1240 to place/move the object 1250 by providing robot state data1238 to the VR rendering module 1210 (e.g., to allow the module 1210 toknow the robot manipulator's location, velocity, and so on as this mayeffect timing of triggering of an interaction event in the VR world bythe module 1210).

The robot control module 1230 functions to generate position/torqueand/or other control commands 1232 that are transmitted to or used tooperate a robot 1240. The robot 1240, as discussed above, may take awide variety of forms to practice the invention and may include amanipulator (e.g., a movable arm and gripper that both can beselectively operated by control module 1230). The robot 1240 is shownwith arrow 1245 to function to position and move (or manipulate) thephysical object 1250 in the space surrounding the user 1204. The controlmodule 1230 takes as input/feedback the joint positions/torques 1234 ofthe robot 1240, which facilitates proper control including generation offuture commands 1232. As discussed above with regard to FIG. 1 and withthe operational example of FIGS. 2-11, the robot 1240 is operated by themodule 1230 to synchronize the movement 1245 to time the delivery of theobject 1250 as well as the velocity, orientation, and/or otherparameters for haptics 1255 with an interaction event produced by the VRrendering module 1210 in video output 1215.

Although the invention has been described and illustrated with a certaindegree of particularity, it is understood that the present disclosurehas been made only by way of example, and that numerous changes in thecombination and arrangement of parts can be resorted to by those skilledin the art without departing from the spirit and scope of the invention,as hereinafter claimed.

The inventors recognized some VR systems had been developed that wereconsidered physical VR systems because they placed real objects inlocations that matched the location of virtual counterparts. In thisway, VR users were able to experience appropriate touch sensations, andthis made the immersion into the VR world self-consistent and morepowerful. However, the inventors understood that such physical VRsystems were only useful for a VR experience that included whollystationary objects, and these systems were not actively controlled tomatch the user's movements.

The inventors with the systems described herein have introduced a VRworld with interaction events in which users of the VR system are ableto have physical interactions with moving or dynamic physical objects orobjects that the user may physically use and move in the real world orsurrounding physical space and have these uses/movements translated intouses/movements of the virtual object that correspond with the physicalobject within the virtual world. The physical objects can moveslow-to-quickly, be made to fly through the space about the user, mayroll, or move in other ways that match those of their virtualcounterparts. Users of the VR system, hence, can both see and touch,catch, grab, toss, hit, or otherwise have physical interactions withvirtual objects in the VR world and concurrently with correspondingphysical objects in the real world about the user. The virtual worlds,thus, become far more dynamic, alive, and interesting to the VR systemuser. The storyline provided by the VR system may allow the user to havea superhero-type experience in which they are able to use theirsuperpowers to have a physical interaction that seems beyond humanabilities (such as to use their mind to cause the physical object to flyacross a space and land in their hand with the virtual object shownflying through space while a robot moves the real world object from anoriginal position to the viewer's hand).

The inclusion of dynamic objects and extensions beyond statically-placedobjects can help create VR worlds that are far richer, more life-like,and, in some cases, superhero-like as virtual reality is mixed with thereal world. The VR experiences are enhanced and can explore more activeinteractions between the user and physical objects and/or virtualobjects in the VR world. The dynamic nature of the physical objects andinteractions with the user also tightens the demand to preciselysynchronize all senses in the VR world including vision, audio, andtouch. The use of a programmable robotic mechanism provides both therequired precision and creates flexibility to achieve various effects.The proposed system(s) taught herein use a robotic mechanism external tothe user that is not attached to or worn by the user. This enables amore natural VR interaction event coordinated with a physicalinteraction outside the VR world as opposed to the experiences providedby previous approaches, which required the user to wear devices forhaptic cues.

The trigger for initiating an interaction event and causing the robot tomove the object from its holding distance or location relative to theuser's hand (or other tracked body part) may vary to practice theinvention. In one embodiment, the user may position their hand (or othertracked body part) in a certain location in the space or with aparticular orientation (and/or location relative to their body) and holdthis pose for a predefined time period. The tracking system tracks thismovement by the user, and the VR system rendering module and robotcontroller may both identify the trigger (or one may identify thetrigger and communicate this to the other controller). Typically, thedistance at which the robot holds the physical object apart from theuser's contact surface/tracked body part will be smaller than (or atleast differ from) the distance in the virtual world between the virtualobject and the representation of the user's tracked body part in thevirtual world. Hence, the robot controller may initiate movement of thephysical object after a time when the virtual object is shown in theimage of the virtual world, with the later time chosen to allow therobot to move the physical object at the same velocity as the virtualobject in the virtual world for simultaneous (or time synchronization)occurrences in the virtual world and the real world of the object(physical and virtual) contacting the user's hand (real world hand andvirtual representation they are viewing in their headset).

In other cases, the virtual world may be generated to mimic movement ofthe physical object by tracking the relative distance between thephysical object and the user's tracked body part and using the samevelocity as the robot to move the virtual object in the virtual world soas to achieve time synchronization of the two contacting events. In somecases, the location of the user's hand/body part is not fixed during theinteraction event and the virtual world is rendered to adjust movementof the virtual object to the new hand/body part location as is the robotin the real world space about the VR user.

It will be recognized by those skilled in the arts that many additionaland different experiences can be generated that blend the physical worldand the VR world through operations of the systems described herein. Forexample, the exemplary operations of the system 100 provided in FIGS.2-11 may be expanded to include one or more extra features. After theoperator physically shows the user the physical object 260 (to establishits reality in the physical world), and before the user puts on thehead-mounted VR display, the operator places the object 260 into anobject (e.g., a tool or weapon) holder, which can be a motor-driven“shaker.” The user is then asked to stretch out their hand to seewhether they can cause the object 260 to leap into their hand from adistance away (e.g., from 10 to 20 feet away). The user stands in thesame place as they will in the future VR case, and the same VR trackingsystem that is employed when the user is immersed in the VR experienceis now used to track their outstretched hand. The user is asked to movetheir hand back and forth slowly to see whether they can attract thephysical object 260.

When the user moves their hand through the area where the robot isoperated/controlled later to hand them the object 260 in the VR world,the object holder rocks the physical object 260 back and forth. Therocking strength is varied by the control system from barely moving torocking back and forth violently as if trying to fly depending on howclose the user's hand is to the future ideal interaction position in thelater VR experience. These “pre-show” operations have two importanteffects: (a) it “trains” the user roughly where their hand should bewhen they will catch in VR; and (b) it also makes the user feel that a“magical” experience was actually occurring (this being in theme withthe overall story line). After the user has tried unsuccessfully to pullthe physical object 260 to them without the HMD on, they are advisedthat their powers will be increased by dawning the HMD helmet, which maybe disguised as a helmet from a particular movie or animation to fit theVR experience and/or otherwise themed to enhance the overall experience.

The systems described herein such as system 100 may be configured to beable to sense user/player actions both when they are in the VR world andbefore they enter it. For example, in a telekenesis simulation use forthe system 100, the object to ultimately be interacted with in the fullVR experience may sit on a motorized platform (or yet another “robot”)so that the user/player is surreptitiously taught how to hold their hand(or other body part(s)) in an area/space that will support the properuse of the ensuing VR experience. The mechanized physical experienceadds credence to the subsequent VR interaction.

With regard to there being a physical object and handover, besidespassing to and grabbing an object from the user, the systems can beconfigured for “sword fighting” or “slapping” where there is a dynamicphysical interaction (and hence, some physical thing), but the objectmay not be detachable from the mechanism. For example, a VR character'shand touches the user and the physical robot also touches the user'shand using some form of paddle. With regard to matching, the physicaland virtual objects do not have to match with regard to: (a) theirmotion, e.g., could take a different path; (b) their speed (especiallyif they take a different path); and (c) their shape, color, texture,details, and the like.

In many implementations, the “dynamic physical interaction” is the coreaspect. In particular, claims calling out “virtual corresponding tophysical” and “move physical based on virtual” should be construed asbeing non-restrictive or be broadly construed, with the physicalenvironment/mechanisms being programmed based on other things, too, insome cases. In many situations, the velocities of the virtual andphysical objects do not have to match (and generally do not). Therelease of the physical object could also happen slightly before or atthe same time as impact. Shortly after allows for forces to build up andgives the object and extra “umph,” which makes sense if the physicalvelocity is smaller than the virtual.

Again, with regard to matching, the virtual and physical objects may bein different places. For example, if the VR participant drops a physicalobject, the virtual object could fall, bounce, or break while thephysical is retrieved. The travel paths can be different for the virtualand physical objects. For example, the virtual flies in an arc while thephysical is delivered in straight line. The velocities may also bedifferent. With regard to impact, the robot may deliver or apply acertain force before releasing such as to simulate different impactspeeds or convey more/less inertia. The robot's release of the physicalobject may be before (so object literally flies into contact) to after(to build up force and assure proper “catching”) while, in otherimplementations, there may be no release (slapping, sword fighting, orthe like) or a grab. With regard to coordination, the coordination canbe determined/driven by virtual (to match the “story”) or physical (toaccommodate last second user movements). Both virtual and physicalsystems/assemblies can drive the coordination, e.g., orientation byvirtual, timing by physical, and the like. Orientation could match theuser's contact surface (palm of hand) or whatever is appropriate to theVR story. The events could be triggered by voice command as well asactions, movements, buttons, and the like.

With regard to the system setup, the “robot” could be cables and/orcould be retractable. The robot could be hidden in the floor, ceiling,walls, and the like. The tracking could be marker-less, and, in somecases, the room could be dark (to further hide things) and trackingprovided with infrared devices. With regard to the physical object(s),the objects used could also be balls, frisbee, sticks, and so on. Thephysical object can include buttons/switches that trigger physicaleffects (vibrations and so on) as well as virtual effects (lights,behavior, extending energy beam, and the like). Again, synchronizationand/or matching makes this exciting for the user. Objects could have a“mind of their own” and act independently of the user. The robot couldbe the physical object.

While the above description stresses the use of the physical interactionsystem (and related processes) with VR systems, those skilled in the artwill readily understand that it may be used in other applications toenhance physical interactions including handoffs or handovers of objectsto human participants or users of the system. For example, augmentedreality (AR) systems may be used in place of the VR system (such as inplace of system 110 in FIG. 1) or other interaction systems may beenvisioned that do not require a VR or an AR system in which it isdesirable to utilize the physical interaction system and other aspectsof the present description. The AR system is not shown in the figures astypical components will be understood from the discussion of the VRsystems. The AR system may utilize a headset or be head mounted and/orone or more projectors to display or project augmentation content to theuser or participant (or “wearer” in the headset-based implementations).

AR-based systems and non-AR and non-VR systems differ from VR systems inthat it may be desirable to provide a more realistic, e.g., human-like,faster, smoother, or the like, handover or handoff of the physicalobject from the robot and its object manipulator (see elements 178 and179 of FIG. 1) to the human participant or user of the system becausethe human participant or user may be able to see or partially see therobot and/or its object manipulator and object during this handover orhandoff (in contrast to a VR application). Steps may be taken to hide ordisguise the robot or its object manipulator, but the following firstpresents a useful system or method for performing fast handovers with arobot (e.g., a robot character at an amusement park or another usecase). A fast and compelling handover is achieved through the inventors'recognition that small sensorimotor delays can be used to improveperceived qualities of the robot's movements during the handover/handoffof the physical object.

Particularly, a system was designed and prototyped for fast and robusthandovers with a robot (or, interchangeably, “robot character”), and auser study was performed investigating the effect of robot speed andreaction time on perceived interaction quality. The handover system(which can be used in a particular implementation of the physicalinteraction system of the present description such as system 170 ofFIG. 1) can match and exceed human speeds and confirms that users preferhuman-level timing in handovers. The system can be configured to havethe appearance of a robot character, such as with a bear-like head and asoft anthropomorphic hand. In some implementations, the system iscontrolled to use Bezier curves to achieve smooth and minimum-jerkmotions.

Fast timing is enabled by low latency motion capture and real-timetrajectory generation, which may involve the robot initially movingtowards an expected handover location and the trajectory being updatedon-the-fly to converge smoothly to the actual handover location. Ahybrid automaton may be used in some handover system implementations toprovide robustness to failure and unexpected human actions (e.g., by theVR or AR participant or system user). In a 3 by 3 user study, theinventors tested the handover system by varying the speed of the robotand adding a variable sensorimotor delay. The social perception of therobot was evaluated using the Robot Social Attribute Scale (RoSAS).Inclusion of a small delay, for mimicking the delay of the humansensorimotor system, led to an improvement in perceived qualities overboth no delay and long delay conditions. Specifically, with no delay,the robot is perceived by the AR/VR participant or system user as morediscomforting, and, with a long delay, the robot is perceived as lesswarm during the object handover.

As an overview or introduction to the new fast handover system, robotsare starting to interact directly with humans and gradually are startingto become part of daily social interactions, e.g., as helpers,companions, and care-givers. This means that robots are not only beingdesigned to be safe and functional but also to act consistent withnormal and expected human behaviors. In this regard, handing over aphysical object requires little conscious thought for a human, butobject handover is filled with expectations and is seen against alifetime of experiences. For robots, handover presents a relevantexample of a direct interaction with a human, and handover interactionsbetween humans and robots have recently been the source of much study.Particularly, researchers are interested in endowing robots withhandover behaviors that users perceive favorably and as natural,competent, and efficient (e.g., to approach behaviors expected fromanother human).

The inventors studied handover interactions with a non-anthropomorphicrobot and identified timing as the factor with the greatest effect onperceived qualities in a handover, and faster behaviors were preferredover slower ones by human participants in the studies. The inventorshypothesized that participants preferred interactions that were moreefficient, i.e., handovers that required less time. However, all thestudied interactions were significantly slower than typicalhuman-to-human handovers. To further explore this area of robot design,the handover systems described herein are designed to be capable ofexecuting fast and robust bi-directional (human-to-robot androbot-to-human) handovers. The inventors believe that matching andexceeding human timing more readily allows users to anthropomorphize andperceive the robot as part of a normal social interaction (e.g., to bemore human like). AR and VR experiences interact primarily with a user'svisual senses; the present description adds to those interactions basedon a user's sense of touch and sense of timing. In this regard, featuresdescribed herein are useful in wholly physical robot interactions, notjust AR and VR experiences.

As part of an effort by the inventors to encourage such perception, therobot was configured to have the appearance of the torso of a bear-likecharacter in one prototype that featured a head and an anthropomorphichand. This prototype robot 1310 is shown in FIG. 13 duringhuman-to-robot handover operations (and the robot 1310 can also performrobot-to-human handovers) with physical object 1320 (in the form of aring in this example). The robot has a torso 1312 supporting a head 1314and an object manipulator 1316 in the form of a robot arm 1318supporting an operable hand 1319 (with fingers for grasping objects),and clothing/skin 1330 is provided on the robot 1310 to hide or disguisethe underlying mechanical features (e.g., of the robot arm 1318).Rounding out this compelling robotic character 1310, the control systemfor the robot 1310 was designed to use adjusted, minimum-jerk movements(e.g., of the arm 1318 and hand 1319) and was robust towards unexpectedor uncooperative human behaviors by human participant or system user1340 (e.g., an AR or VR participant wearing a headset (not shown in FIG.13 but understood from earlier figures)).

A handover system incorporating the robot 1310 was used to conduct a 3by 3 user study where the inventors varied the speed of the robotmotions and the system reaction time. To vary the reaction time, theinventors included a variable sensorimotor delay in the control of therobot 1310. Closed-loop control with a small sensorimotor delay washypothesized to create a more compelling handover behavior when includedto mimic the latency of the human sensorimotor system. The followingthree levels of sensorimotor reaction time were considered: (1) no delay(faster-than-human reaction); (2) short delay (similar to human reactiontime); and (3) long delay (slower than human reaction time). For thespeed, the inventors considered the following three levels: (1) slow;(2) moderate; and (3) fast, with the moderate speed or condition beingsimilar to the speed of human arm motions. The inventors' study resultsshowed that the inclusion of the short sensorimotor delay improves theperceived qualities of the robot during handover operations. With nodelay added, the system is perceived by human as more discomforting,independent of the arm speed conditions tested, and, with a long delay,the robot is perceived as being less warm during handover operations.

Turning now to the system for fast and robust handover interactions, thefollowing description considers the handover task to be bidirectionalhandover interaction. The human initiates the handover in some cases bypresenting the object. The robot is controlled by its controller toreach, grasp, and move the object from the handover location to itsresting position. It then is controlled to return the object to the samehandover location to handover the object back to the human participant.

FIGS. 14A-14F depict the handover sequence as carried out by a handoversystem that includes the robot 1310 being controlled to provide handoverof a physical object 1320 to a human participant 1340. The handoversequence includes: (a) in FIG. 14A, a handover step 1410 in which therobot 1310 and the human 1340 are both in rest poses with the object1320 placed in a cradle close to the human's right hand; (b) in FIG.14B, a handover step 1420 that involves the human 1340 presenting theobject 1320 to the robot 1310; (c) in FIG. 14C, a handover step 1430that includes the robot 1310 reaching out and grasping the object 1320;(d) in FIG. 14D, a handover step 1440 that includes the robot 1310returning with the grasped object 1320 to the rest pose; (e) in FIG.14E, a handover step 1450 that involves the robot 1310 presenting theobject 1320 to the human 1340; and (f) in FIG. 14F, a handover step 1460that includes the human 1340 grasping the presented object 1320. Afterthese steps, both the robot 1310 and the human 1340 return to their restposes, and the object 1320 is returned to the cradle. Note, in theimplemented or prototyped handover system, the robot 1310 does not waitfor the object 1320 to reach the handover location before it startsmoving nor does the human 1340.

FIG. 15 illustrates the physical object 1320 as used in testing thehandover system of the present description in a support cradle 1570prior to and after handover operations. In one prototype of the handoversystem, the object 1320 took the form of a toroidal object with a 30-cmouter diameter, an 18-cm inner diameter, and a 4.5-cm thickness. Thetoroidal or ring shape was chosen for the object 1320 so that it couldbe readily grasped by the robot 1310 with its hand 1319 (and humanparticipant 1340) from a range of approach angles. This shape for object1320 is also useful as it distances the human 1340 from the robot hand1319 during handover when both are grasping the object (see steps 1430and 1460 of FIGS. 14C and 14F), minimizing potential interference andimproving safety. In one embodiment of the handover system with therobot 1310, tracking was provided with an OptiTrack motion capturesystem, which used a constellation of retroreflective markers 1550 onthe toroid object 1320 to track its position and orientation. Users1340, in this example in contrast to earlier examples, did not wearmarkers or other instrumentations (but, they may in some otherimplementations). During the user study/prototype testing, the object1320 was initially placed in a cradle 1570 in reach of the humanparticipant 1340. The handover interaction started (control over therobot's movement were initiated for handover) when the object 1320 wasremoved (by the human 1340 or robot 1310) from the cradle 1570.

With regard to the robot's physical design for use in the handoversystem, a robot 1310 was designed and built to aid the perception of therobot 1310 as a social entity. In one prototyped handover system, therobot 1310, as shown again in FIG. 16, was created to have theappearance of an anthropomorphic bear-like character with torso 1312, anarm 1318, and a head 1314. The prototyped handover system used a KUKALBR iiwa 7 R800 robot for robot 1310, which was mounted as shownhorizontally onto a support frame 1680 to form the shoulders and rightarm 1318 of the bear character. In this way, joint 6 of the robot 1310becomes the elbow of the character in arm 1318, and joint 4 becomes thecharacter's right shoulder. A cartoon bear head was used for head 1314and attached to the second link, and the torso 1312 was dressed in ashirt 1330 to reinforce the illusion of a character handover. Joints 1and 2 of the robot 1310 controlled the pitch of the head 1314 and reachof the right shoulder, respectively.

To implement the hand 1319, a Pisa/HT SoftHand was used in the robot1310 to provide a soft and underactuated hand with a single actuateddegree of freedom. The hand has an anthropomorphic appearance, whichsupports the character's appearance. Moreover, the softness of this typeof robotic hand for hand 1319 allows the hand 1319 to robustly grasp inthe presence of small locational variations of the object 1320. Toenhance the character behavior, joint 1 of the robot 1310 tilts the head1314 to appear to look at the object 1320 during handover steps. Joint 2of the robot 1310 allows the character to lean forward with its rightshoulder to reach towards a more distant handover location, e.g., beyonda specified radius of its right shoulder, or to lean back if thelocation is too close. An analytic inverse kinematics solver was used inthe robot controller to compute the remaining five joints to grasp thetoroid 1320 with the hand 1319. With two axes of symmetry around themajor and minor radii, the toroid 1320 strictly requires only fourdegrees of freedom to achieve a grasp by the robot 1310. Thus, theinventors used the final degree of freedom, via control of the robot1310, to keep the elbow height in arm 1318 as low as possible duringhandover.

With regard to online trajectories for fast and smooth motions, it wasrecognized by the inventors that human receivers often begin reachingfor the object before it has reached its final handover location. Toenable this behavior in the robot, the inventors introduced the notionof an expected handover location, x_(exp). The position of thispredefined location was chosen to be approximately half way between therobot 1310 and the human participant 1340 and slightly to the right-sideof the robot 1310 (for a righthanded handover and vice versa for alefthanded handover). The initial location for x_(exp) may be determinedbased on characteristics of the human receiver as well, such that anx_(exp) might start differently for a child as compared to an adult, ora receiver who is sitting versus standing. The smooth, minimum-jerktrajectory was precomputed to the joint configuration at this locationbased on Bezier curves. Execution is initiated by the robot controller(or handover system controller in some cases) in response to theappropriate trigger. During movement of the robot 1310 (including itsarm 1318), the trajectory is updated on-the-fly as the object 1320 ismoved such that the robot 1310 converges to the object location. Thisallows the robot 1310 to begin motion as soon as the human 1340initiates the interaction (by picking up and presenting the object 1320in this example).

Specifically, when the object 1320 is removed from the holding cradle1570 and after a specified reaction time, the robot 1310 is controlledto initiate the precomputed trajectory, q_(pre)(t), having a duration oft_(f) towards the joint configuration q(x_(exp)). After an initialduration of td, a new target, q(x_(obj)(t)), is continually computed,applying the inverse kinematics to the current object location,x_(obj)(t). A gradually increasing fraction of the equivalent shift issummed to the precomputed movement. This process of calculating theonline joint trajectory, q(t), is portrayed in FIGS. 17A-17C withschematics 1710A-1710C and graphs 1720A-1720C and with the followingequation:(q(x_(obj)(t)−q(x_(exp)))

By the conclusion of the trajectory, the robot 1310, thus, reaches theobject 1320. Overall, this accomplishes a smooth, natural-looking motionthat appears both predictive (moving early) and responsive (watching theobject's placement). The prototyped implementation in a handover systemwith robot 1310 ran in real time at 1 kHz (referring to the real-timerobot control loop). Regarding real-time control of the robot, in orderto drive force/torque interactivity of the robot system, the robotcontroller runs on a real-time loop to guarantee that the robot canrespond to sensor inputs with minimal latency. Take, for example, acompliant robot where if someone pushes on the robot, the robot sensesthis force and gives way with velocity or position proportional to themagnitude of the force. If hard real-time is not guaranteed, what mighthappen is someone pushes on a robot that feels rigid, and then gives wayhalf a second later which would not make for convincing compliantbehavior. For the prototyped robot system, the inventors ran the robotcontrol loop at 1 kHz to maintain responsiveness of the robot toexternal inputs.

As shown in FIG. 17A during the interactive tracking method, the robotinitially moves along a precomputed trajectory towards an expectedhandover location, x_(exp). In FIG. 17B, after an initial duration, td,of the arm motion is complete, a new joint target q(x_(obj)(t))representing in joint space the current object location, x_(obj)(t) iscomputed by the robot (or system) controller. The control algorithm thenshifts the precomputed trajectory proportional to the remaining time.Then, as shown in FIG. 17C, the trajectory is continuously updated inreal time, as the object moves through the workspace, and this ensuresthe robot smoothly reaches the object position, x_(obj)(t_(f)).

In some preferred embodiments of the handover system and robotcontroller, the handover interaction behavior of the robot is determinedby a hybrid automaton, which enables the system to respond in a robustmanner to unexpected or uncooperative human behaviors. FIG. 18 shows aflowchart for an exemplary human-to-robot handover 1800 (or itscorresponding robot control method implemented by a robotcontroller/handover system controller). The method 1800 starts at 1810with the controller placing the robot in a rest pose. At 1820, tracking(or other techniques) are performed by the system to determine whetherthe object has been removed from the cradle, and, if not, the step 1820is repeated. If removed, the method 1800 continues at 1830 with thecontroller causing the robot to start to move (or its arm or otherobject manipulator) toward the expected handover location (e.g., using afirst part of a predefined/precalculated trajectory). At 1834, thecontroller provides control signals to cause the robot to continue toexecute the smooth trajectory, which it updates in real time to convergeto the true object location (based on processing of tracker outputregarding the actual position of the object (e.g., in a humanparticipant's hand or the like)).

The method 1800 continues at 1840 with the controller determiningwhether or not the object is now inside the robot's hand. If not, themethod 1800 continues at 1846 with the robot being controlled to backaway and to retry by repeating step 1834. If the object is determined instep 1840 to be in the robot's hand, the robot is operated at 1850 toclose the hand to grasp the object with its fingers. At 1860, the method1800 then continues with starting the arm retraction.

During retraction, the controller acts at 1870 to determine if theobject has remained in the hand (still grasped). If not, the controllercontrols the robot to open its hand (or otherwise return to apre-grasp/handover pose) at 1874 and then at 1890 to return to the restpose. If the object is still in the hand at 1870, the method 1800continues at 1880 with determining whether or not the human participantis applying force (still grasping/holding the object). If not, themethod 1800 continues at 1890 with the robot being operated to return tothe rest pose with the grasped object. If yes at 1880, the method 1800continues first with the controller acting at 1886 to reduce armimpedance, slowing down trajectory, and waiting for the humanparticipant to release the object before moving to step 1890.

As shown with the method 1800, if the grasp fails in the handoverprocess and the object moved a small amount, the robot will try to reachfor the object again. When retreating from a successful grasp, the robotuses impedance control with low stiffness to accommodate any humanforces. Only when the human participant/system user has released theobject will the robot be operated to fully return to the rest pose.

FIG. 19 illustrates a robot-to-human participant handover 1900 thatbegins at 1910 with the robot grasping the physical object while beingpositioned in its rest pose (e.g., with arm holding the object hangingvertically downward from the torso). At step 1920, the robot is operatedto move its arm (object manipulator) to move the object to a predefinedor calculated handover location. Then, at 1930, the controllerdetermines whether or not the human participant is applying a force onthe object. If not, there will not be a successful handover of theobject, and the method 1900 continues at 1940 with waiting a period oftime (such as 2 to 5 seconds or the like) while repeating step 1930.When the time elapses without a force being detected, the method 1900continues at 1950 with shaking the object to signal to the humanparticipant to take the object (or, in some cases, with a return to step1910 by moving back into the rest pose) and then repeating step 1930.

If a force is detected in step 1930 (e.g., a force over a predefinedminimum handover limit), the method 1900 continues at 1960 with therobot being controlled to open its hand so as to release its grasp onthe object. The handover process 1900 is completed then at step 1970with the robot being operated to return (empty handed) to its rest pose.As shown in FIG. 19 (illustrating an exemplary hybrid automaton), duringthe robot-to-human handover 1900, the object is released when the humanforce exceeds an appropriate threshold. Testing of the two methods 1800and 1900 have shown that the handovers achieved with the robot 1310 andthe handover system including this robot 1310 are fast (have desirablespeed) and also robust.

Using the handover system described above, a study was designed andperformed to investigate the effect of robot speed and reaction time onthe perceived qualities of the interaction. The inventors hypothesizedthat matching human characteristics would improve the experience forhuman participants and that excessive speeds may actually prove to becounterproductive. With regard to the testing method, a 3 by 3experimental design was implemented to investigate the effect of robotspeed and reaction time. For the robot's speed, the three conditions ofslow, moderate, and fast were chosen by visually comparing therobot-to-human handover motions and adjusting parameters to obtain aspeed similar to what a human would commonly used (i.e., a moderatespeed for the test for controlling the movement of the robot's arm) withslow and fast speeds being defined relative to this moderate speed. Theanalysis below compares the speeds of the human and robot in theexperiments.

To vary the reaction time of the system, the inventors artificiallyinduced delays in the robot's reaction times to stimuli (e.g., to motioncapture and force data). The three conditions of no delay (e.g.,0.03-second latency), short delay (e.g., 0.25-second latency), and longdelay (0.75-second latency) were tested. The short delay condition has adelay that is similar to the delay of the human sensorimotor system,whereas the no delay condition enables faster-than-human robot responseand represents the fastest achievable reaction time of the system.

Eighteen human participants (9 male and 9 female) with ages ranging from21 to 41 (M=29,82, SD=5.88) completed the study. Participants were askedto stand in a designated area in front of the robot, with a distance ofapproximately 140 cm from the robot. The object was placed in a cradlelocated to the right of the participants. At the start of each segmentof the study/trial, the experimenter verbally signaled to theparticipant to pick up the object from the cradle and hand it over tothe robot. The robot then was operated to retrieve the object from theparticipant and bring its arm back to the rest pose with the object inits hand for 0.5 seconds. Then, the robot was controlled to proceed tohand the object back to the participant at the same location where thehuman-to-robot handover took place. The participant retrieved the objectfrom the robot and returned it to the cradle to conclude that segment ofthe study/trial.

Each participant completed four trials per condition, and each consistedof a human-to-robot and robot-to-human handover (for a total of 36human-to-robot and 36 robot-to-human handovers per participant). Theorder of the conditions was counterbalanced using a Williams designLatin Square to mitigate first-order carryover effects. After eachcondition, participants were asked to complete a RoSAS questionnaire,which provided a set of eighteen Likert-scale questions pertaining tothe three underlying factors of warmth, competence, and discomfort (with7-point Likert scales used). Throughout the experiments, the pose of therobot (in both joint and Cartesian coordinates) and object and also thetimes that the object was removed from and replaced back into the cradlewere recorded for later analysis. The experiment lasted about 45 minutesper participant.

Turning to results, with regard to the RoSAS, a two-way repeatedmeasures MANOVA was conducted to test the effects of robot end-effectorspeed and reaction time on the RoSAS attributes. Effect sizes in termsof partial eta squared, IV, are reported. As a rule of thumb, it hasbeen indicated that partial eta square values of 0.0099, 0.0588, and0.1379 may serve as benchmarks for small, medium, and large effectsizes. Post hoc pairwise comparisons were adjusted for multiplecomparisons using the Bonferroni correction.

A significant main effect of speed on reports of discomfort was detected[F(2,32)=4.483, p=0.019, n_(p) ²=0.550]. Post hoc pairwise comparisonsfound that the average discomfort score for the fast speed [M=2.529,SD=0.928] is 0.425 points higher than the slow speed [M=2.105,SD=0.870]. Significant main effects of reaction time on warmth[F(2,32)=4.561, p=0.018, n_(p) ²=0.222] and [F(2,32)=4.369, p=0.021,n_(p) ²=0.215] discomfort were also found. Post hoc pairwise comparisonsindicate that the average warmth score for the short delay reaction timeis 0.333 points higher than the long delay reaction time [p=0.020],representing a small effect size [d=0.286]. In terms of discomfort, theno delay reaction time scored higher than both the short delay and longdelay reaction times by 0.252 [d=0.286, p=0.044] and 0.242 [d=0.274,p=0.023] points, respectively. No other significant main or interactioneffects were detected at the α=0.05 level. Mean ratings and significanteffects are tabulated in Table I and II.

TABLE I ROSAS ESTIMATED MARGINAL MEANS AND STANDARD ERRORS FOR ROBOTSPEED. warmth competence discomfort speed mean std. err. mean std. err.mean std. err. slow 3.520 0.350 4.732 0.250 2.105 0.211 moderate 3.4250.305 4.827 0.267 2.275 0.249 fast 3.310 0.274 4.742 0.255 2.529 0.225

TABLE II ROSAS ESTIMATED MARGINAL MEANS AND STANDARD ERRORS FOR ROBOTREACTION TIME. warmth competence discomfort reaction mean std. err. meanstd. err. mean std. err. no delay 3.229 0.245 4.680 0.235 2.225 0.220short delay 3.562 0.315 4.807 0.261 2.216 0.239 long delay 3.464 0.3054.814 0.266 2.467 0.202

Turning now to a comparison of human and robot arm speeds, the inventorsused the motion capture data from the test to compute the average speedof the object when it was being moved by the human participant as wellas the average speed of the object when it was being moved by the robotfor three different speed conditions. The mean peak velocity was alsocomputed across all trials for both the human and the three robot speedconditions. The results of these computations are provided below inTable III.

TABLE III AVERAGE AND PEAK SPEEDS ACROSS HUMAN PARTICIPANTS AND FORDIFFERENT ROBOT SPEED CONDITIONS. average standard peak standard speed(m/s) deviation speed (m/s) deviation Human 0.638 0.119 1.355 0.549Robot (slow) 0.369 0.054 0.507 0.090 Robot (moderate) 0.573 0.054 0.8820.101 Robot (fadt) 0.694 0.069 1.149 0.123

The obtained average speed for the human participants was somewhatfaster than the value of 0.55 m/s reported in some prior handoverstudies. It can be seen that the average speed for human participants inthe inventors' experiments lied between the moderate and fastconditions. One can also observe that the human motion has a greatervariability in speeds, i.e., a greater difference between the averagespeed and the peak speed than the robot conditions. This would beexpected due to the significantly smaller inertia of the human arm. Theanalysis shows that the three speed conditions studied were indeedsimilar to average human arm speeds.

The inventors' study of their new robot handover system showed that itadvantageous to mimic a human-like sensorimotor delay for human-robothandover interactions. With a longer delay, the robot is perceived byparticipant as less warm, but, with no delay, the robot is perceived asmore discomforting. This suggests that sensorimotor delays are moregenerally beneficial and could be applied to other interactive roboticsystems and also digital characters. The inventors also discovered thatfast movements are perceived as discomforting, while there as noobserved difference between the slow and moderate speed conditions forthe robot. Considering the values in Table III, this indicates thatparticipants prefer the robot moving slower than, or perhaps at, theirown speed. The fact that no effect on competence was observed in thestudy suggested to the inventors that all conditions were sufficientlyfast or sufficiently close to human speeds.

As seen from the analysis on movement speeds, the robot handover systemis able to match human handover speeds. This is highly relevant forbeing able to observe the effects seen as part of the study because asthe system behavior is closer to human behavior, participants are ableto more readily apply human social expectations to interpret the robotbehavior. The character-like appearance of the robot would be expectedto further support this transfer of social expectations.

The above describes a fast and robust system for handovers with a robotcharacter. Fast handovers are achieved, at least in part, by an onlinetrajectory generator (e.g., software run by the robot and/or systemcontroller) that allows the robot to begin motion as soon as the humaninitiates a handover (e.g., movement of the object toward the expectedhandover location is detected). The generator also smoothly adjusts therobot's trajectory as the object moves and converges to the objectlocation, giving the robot's reaction (e.g., modification of the arm'stravel path) a natural and responsive feel. Robustness is achieved, atleast in part, by a hybrid automaton that detects system failures, suchas the human being uncooperative, and that responds accordingly. A fastrobot motion capture system was included in the system that had minimallatency, and the handover system also benefited from use of a soft handand design of the robot to provide an inviting appearance (e.g., abear-like character). The combination of these system aspects helped tocreate a robot interaction that felt to the human participants in theobject handover as being organic and character-like (or, in general,human-like).

The above handover system and methods may be useful in ARimplementations or systems that include a physical interaction system asthe AR participant can observe the surrounding physical space includingthe robot and its object manipulator used to perform the handover. Thehandover performed by the robot (such as robot 1310) may be synchronizedwith provision of augmentation content (similar to the providing of VRinteraction and content with synchronization of an object manipulator).

For example, the AR headset may be operated to provide augmentationcontent that augments the physical object (e.g., so that its appearanceis changed, so that information regarding the object is provided, and soon) and/or the robot or its manipulator. In some cases, a virtualcharacter or other digital artwork/content may be superimposed by the ARheadset upon the robot and/or its object manipulator (e.g., a roboticarm with a hand for grasping and releasing physical objects) so that therobot's appearance is changed to suit the AR world or experience. Suchsuperimposition will be performed in real time by tracking the locationof the robot and its components and the location and orientation of thephysical object as well as the AR participant's location and viewingangle/direction (as discussed above for tracking VR participant) and byadjusting the augmentation content displayed by the AR headset toprovide accurate superimposition upon the robot and/or physical object.In some cases, the augmentation content may be used to block viewing ofthe robot and its arm/object manipulator by the AR participant such asthrough the use of opaque AR techniques. Further, in place of an ARheadset, the system may utilize projection-based AR to provide similaraugmentation content to the AR participant.

With the new physical interaction system and techniques, unique visualeffects can be achieved in VR and AR applications and environments. Thevisual effects and displays may be provided through head-mounted VR/ARdevices and/or may use projections, lighting, and/or visual illusions inAR-based systems to achieve desired effects (such as augmenting thephysical object, costuming, hiding, or otherwise modifying therobot/object manipulator, and the like). In some cases, costuming ormechanically hiding robot the robot may be used in the AR and/orVR-based systems to further a designed illusion or storyline. Forexample, a robot in costume to look like a person (e.g., a particularcharacter from a movie or the like) handing physical objects to a personin the physical interaction system. In some embodiments, the robot maybe out of view (e.g., reaching up or down (or laterally) into viewablespace to move object. In some cases, the robot and/or its objectmanipulator may be made invisible or nearly so to an VR or ARparticipant such as by use of limited lighting, projectedlighting/augmentation content, and/or through the design of the objectmanipulator (e.g., manipulated wires or strings or the like to moveobjects in the space).

The new physical interaction system can be operated and/or controlled toprovide unique timing and travel paths for the physical object. Forexample, an AR object can become a real object when a light turns on.Paths of travel for the object may be physical or hyper (e.g., aninvisible robot may move a real object in non-physical ways such as bymoving slower or faster than expected or along a path not expected by areal world object). The movement of the physical object can be adjustedto movement of user/participant, e.g., deliver object when user isreaching for it, make user chase (reach for) object, and the like. Thephysical object can have its weight or heft varied by the robot/objectmanipulator such as by varying forces at contact to make the objectappear more or less massive than it actually is in the real world. Therobot can be operated (as discussed, for example, with the handoversystem) to not only deliver an object but to take the object back fromthe participant. The physical object can be of practical use in the VRor AR world (e.g., can reversely affect the augmented/virtual worlds).In some cases, the handover robot (or other robot with an objectmanipulator) may be controlled not to simply provide an object but toprovide other physical interactions with the participant such as a swordor other fight in which a physical object strikes or contacts theparticipant's body or an object they are already holding.

We claim:
 1. A system for providing a dynamic physical interaction to ahuman participant, comprising: a VR system including a headset with adisplay screen and a VR rendering module generating a video output,wherein the display screen displays an image of a virtual world based onthe video output; and a physical interaction system comprising a roboticmechanism and a robot control module generating commands for operatingthe robotic mechanism to selectively position a physical object within aspace, wherein a wearer of the headset is positioned in the space,wherein the image of the virtual world includes an image of a virtualobject corresponding to the physical object, and wherein the commandsgenerated by the robot control module cause the robotic mechanism tomove the physical object in the space based on a state of the virtualobject in the virtual world to cause a physical interaction between thephysical object and the wearer in a time synchronized manner suited tothe state of the virtual object in the virtual world.
 2. The system ofclaim 1, wherein, after the physical interaction, the VR renderingmodule generates the video output to modify the image of the virtualworld to show an effect of the wearer having or using the virtual objectin the virtual world.
 3. The system of claim 1, wherein the roboticmechanism releases the physical object prior to the physicalinteraction.
 4. The system of claim 1, wherein the physical interactioncomprises a handover of the physical object to the wearer.
 5. The systemof claim 4, wherein the commands generated by the robot control moduleafter the physical interaction cause the robotic mechanism to retrievethe physical object from the wearer.
 6. The system of claim 1, whereinthe commands generated by the robot control module prior to the physicalinteraction cause the robotic mechanism to modify a velocity, a travelpath, or a position of the physical object in response to movements ofthe wearer in the space.
 7. The system of claim 1, wherein the commandsgenerated by the robot control module cause the robotic mechanism to setforces applied by the physical object on the wearer to increase ordecrease inertia of the physical object perceived by the wearer.
 8. Thesystem of claim 1, wherein the commands generated by the robot controlmodule cause the robotic mechanism to move the physical object intocontact with a surface on a body part of the wearer at a first time,wherein the image of the virtual world includes an image of a virtualbody part corresponding to the body part of the wearer, and wherein theimage of the virtual object is shown in the image of the virtual worldto move into contact with the image of the virtual body part at a secondtime related to the first time.
 9. The system of claim 8, furthercomprising a tracking system generating tracking data for the body partof the wearer, wherein the image of the virtual body part is provided ata location in the image of the virtual world corresponding to a bodypart location identified by the tracking data and wherein the roboticmechanism moves the physical object to a location in which the physicalobject at least partially coincides with the body part location to causethe physical object to contact the body part of the wearer at the firsttime.
 10. The system of claim 9, wherein the tracking data includes anorientation of the body part of the wearer and wherein the commandscause the robotic mechanism to adjust an orientation of the physicalobject based on the orientation of the body part of the wearer prior tothe first time.
 11. The system of claim 8, wherein the body partincludes a hand of the wearer of the headset, wherein the surfacecomprises the palm of the hand, and wherein the commands cause therobotic mechanism to halt travel at the location or at a preset distancepast the location at about the first time.
 12. The system of claim 1,wherein the state of the virtual object in the virtual world includes avelocity and wherein the commands cause the robotic mechanism to movethrough the space at a velocity related to the velocity of the virtualobject in the virtual world.
 13. The system of claim 1, wherein thecommands generated by the robot control module cause the roboticmechanism to release the physical object at a third time after the firsttime.
 14. A system for providing a dynamic physical interaction to ahuman participant, comprising: an augmented reality (AR) systemdisplaying or projecting augmentation content visible by a participantin a space; a physical object positioned in the space; and a physicalinteraction system comprising a robot with an object manipulator and acontroller generating control signals to operate the robot, and whereinthe control signals cause the object manipulator to perform a handoverof the physical object to or from the participant.
 15. The system ofclaim 14, wherein the object manipulator moves toward or away from anexpected handover location in the space at a speed matching or less thana predefined handover speed for a human performing handovers.
 16. Thesystem of claim 14, wherein the controller initiates the handover apredefined delay period after sensing initiation of the handover by theparticipant.
 17. The system of claim 16, wherein the predefined delayperiod is set at sensorimotor reaction time for a human performinghandovers.
 18. The system of claim 14, wherein the control signals causethe object manipulator to travel over a predefined smooth trajectorytoward the expected handover location and wherein the controllermodifies the travel of the object manipulator prior to the handoverbased on tracking data for the physical object, whereby the objectmanipulator converges on an actual location of the physical object inthe space.
 19. The system of claim 14, wherein the handover is arobot-to-participant handover and wherein the controller generates thecontrol signals to cause the object manipulator to release the physicalobject only after sensing of a force greater than a minimum forcethreshold is being applied upon the physical object.
 20. The system ofclaim 14, wherein the handover is a participant-to-robot handover andwherein the controller generates the control signals to cause the objectmanipulator to return to a rest pose only after grasping the physicalobject and sensing the participant has released the physical object. 21.The system of claim 14, wherein the augmentation content is superimposedupon at least the object manipulator of the robot to modify appearanceof the robot to the participant or to at least partially block ordisguise viewing of the robot in the space by the participant.
 22. Thesystem of claim 14, wherein the system includes a tracking systemtracking a location of the physical object in the space and wherein theaugmentation content is superimposed upon the physical object tovisually augment the physical object as the physical object is movedabout the space based on the tracked location.
 23. A system forproviding a dynamic physical interaction to a human participant,comprising: a robot comprising a robotic arm and a hand coupled to therobotic arm; and a controller generating control signals to operate therobot, wherein the control signals cause the robotic arm to perform ahandover of a physical object to or from human participant, and whereinthe robotic arm moves toward or away from an expected handover locationin the space based on a predefined smooth trajectory, and wherein thecontroller initiates the handover a predefined delay period aftersensing initiation of the handover by the participant.
 24. The system ofclaim 23, wherein the predefined delay period being set at sensorimotorreaction time for a human performing handovers and wherein the controlsignals cause the robotic arm to move toward or away from the expectedhandover location at a speed matching or less than a predefined handoverspeed for a human performing handovers.
 25. The system of claim 24,wherein the controller modifies travel of the robotic arm prior to thehandover based on tracking data for the physical object, whereby thehand of the robot converges on an actual location of the physical objectin the space.
 26. The system of claim 24, wherein the handover is arobot-to-participant handover and wherein the controller generates thecontrol signals to cause the hand to open to release the physical objectonly after sensing of a force greater than a minimum force threshold isbeing applied upon the physical object.
 27. The system of claim 24,wherein the handover is a participant-to-robot handover and wherein thecontroller generates the control signals to cause the robotic arm toreturn to a rest pose only after grasping the physical object with thehand and sensing the participant has released the physical object.